4.1 KB · ⇊ 3 parts · ⤓ Micron 4.1 KB · ⤓ HTML 52.3 KB

top

DeepSpeed is an open source deep learning optimization library for PyTorch.cite-ref-1[1]

• Library

• See also

• References

• Further reading

• External links

The library is designed to reduce computing power and memory use and to train large distributed models with better parallelism on existing computer hardware.cite-ref-2[2]cite-ref-3[3] DeepSpeed is optimized for low latency, high throughput training. It includes the Zero Redundancy Optimizer (ZeRO) for training models with 1 trillion or more parameters.cite-ref-4[4] Features include mixed precision training, single-GPU, multi-GPU, and multi-node training as well as custom model parallelism. The DeepSpeed source code is licensed under MIT License and available on GitHub.cite-ref-5[5]

The team claimed to achieve up to a 6.2x throughput improvement, 2.8x faster convergence, and 4.6x less communication.cite-ref-0-6-0[6]

• Comparison of deep learning software

• Deep learning

• Machine learning

• TensorFlow

cite-note-11. ↑ "Microsoft Updates Windows, Azure Tools with an Eye on The Future". PCMag UK. May 22, 2020.

cite-note-22. ↑ citerefyegulalp2020Yegulalp, Serdar (February 10, 2020). "Microsoft speeds up PyTorch with DeepSpeed". InfoWorld.

cite-note-33. ↑ "Microsoft unveils "fifth most powerful" supercomputer in the world". Neowin. 18 June 2023.

cite-note-44. ↑ "Microsoft trains world's largest Transformer language model". February 10, 2020.

cite-note-55. ↑ "microsoft/DeepSpeed". July 10, 2020 – via GitHub.

cite-note-0-66. ↑ "DeepSpeed: Accelerating large-scale model inference and training via system optimizations and compression". Microsoft Research. 2021-05-24. Retrieved 2021-06-19.

• citerefrajbhandarirasleyruwasehe2019Rajbhandari, Samyam; Rasley, Jeff; Ruwase, Olatunji; He, Yuxiong (2019). "ZeRO: Memory Optimization Towards Training A Trillion Parameter Models". arXiv:1910.02054 [cs.LG].

• AI at Scale - Microsoft Research

• GitHub - microsoft/DeepSpeed

• ZeRO & DeepSpeed: New system optimizations enable training models with over 100 billion parameters - Microsoft Research